

Overview 
of Findings 


The good 

• Most first time users could see themselves using the Wayback Machine 
again, primarily for research. 

• More technically advanced users enjoyed the Explore graph and were 
pleased to be able to explore the captures by event type (major and 
minor changes, 404's, redirects, etc). 

• Users enjoyed the 'type to search' feature on the search bar, although 
most users also expected to see a separate result page. 

The not so good 

• First time users, especially those with limited technical expertise, had 
trouble deciphering the terms used on the Explore graph, and were 
unsure of how the graph related to the website they had searched for. 
Even for more advanced users 'minor' vs. 'major' changes was unclear. 

• New users had a hard time understanding what the Wayback Machine 
was or how it could be used and needed more guidance around the 
potential uses of and limitations of the system. 

• The look and feel of the Wayback logo and Ul felt dated to users. 



Overview 
of Findings 


High-level themes 

• New users had a hard time understanding what the Wayback Machine 
was or how it could be used. Some users thought that it was a search 
engine, others imagined it was a suite of developer tools and many did 
not understand the difference between captures and screenshots. 

• Users needed more guidance and feedback from the system. Error 
messages, long loading times and general issues with navigation were 
frustrating for users. Users wanted more information about the depth 
and breadth of a capture (which links in the capture work? which days 
are available?) and very few users were able to correctly read redirect or 
error pages. Users also needed more guidance in deciphering the 
Explore and Capture page graphs. 

• The look and feel of the Wayback Machine, especially the logo and 
homepage design, was distracting to users and lead some users to feel 
as though the site was untrustworthy. 

• The global Archive.org navigation bar was confusing and distracting for 
users. The navigation bar often led users to believe that searches could 
be filtered by media type. 



15 new users recruited from Usertesting.com performed 6 
tasks on Wayback Beta through an unmoderated session. 


About 
the Test 


The goals of the sessions were to determine: 

• Can the user successfully decipher what tasks can be accomplished using 
the Wayback Machine? Do they understand what the Wayback Machine is? 

• Can the user successfully locate the search box on the Wayback homepage? 

• Can the user successfully navigate to an 'Explore' page? Once there, can 
the user correctly interpret the Ul elements on the 'Explore' page? 

• Can the user successfully navigate to a 'Capture' page? Once there, can the 
user understand the information presented on the 'Capture' page? 

• Is browsing through capture history both on the 'Explore' and 'Capture' 
pages easy to understand or execute? 

• Is the structure of information easy for users to understand? 

• Is the Wayback Machine enjoyable or frustrating to use? 

• How can the interface be improved? Where are the toughest pain points? 



Watch the sessions on UserTesting.com 

Tests occurred from July 21 - 25, 2016 
Users were recruited from the US, UK and Canada 


Dive-in to 
the Sessions 



Task 

Analysis 


Homepage 

• The 'universal' Archive.org navigation bar was confusing to many users 
who assumed that the icons on the navigation bar could be used to 
filter the Wayback Machine by media type. One user also accidentally 
used the search box for Archive.org and was lead away from her task. 

• Many users had a hard time deciphering what the Wayback Machine is 
or how they could use it from the information presented on the 
homepage. A number of users assumed it was a search engine. 

• The carousel of historical webpages was one of the first elements that 
users noticed when visiting the homepage, however users generally 
had a hard time deciphering the purpose or utility of the thumbnails. 

• The logo felt 'outdated' or 'untrustworthy' to some users. 

• The Archive-it and Tools text attracted the attention of the majority of 
users, however this text lead some users to believe that the Wayback 
Machine was a subscription based product or a suite of developer tools. 

• The word 'citation' in the Save Page Now text lead nearly a third of 
users into having a different view of what the site could be used for (eg. 
personal storage of citations) 



Results Page 

• Overall, users enjoyed using the 'type to search' feature in the search 
bar, and were impressed by not needing to hit 'enter' or click a button 
to submit their search. 

• Users wanted more information about each of the presented results, 
and were unsure of what the text in the right column represented. Users 
generally wanted to know the URL, website title, information about the 
website and if information related to capture scope or the most recent 
capture. Users did like being able to have a thumbnail of the website. 

• Some users were overwhelmed by the amount of search results and 
were unsure of which was the most relevant or best result based on 
their query. 


Task 

Analysis 



Explore Graph 

• The majority of users did not expect to see a graph after selecting a 
result. A spark graph for each result on the result page or additional 
information related to captures might help to prep users for the Explore 
page graph. 

• New users, especially those without technical expertise had a hard time 
deciphering the labels on the Explore graph. Users who were able to 
understand what the majority of the graph labels meant still had trouble 
differentiating between minor and major changes. Additional language 
around the labels via a tool tip would help users better understand and 
utilize the Explore page graph. 

• Users had a hard time differentiating between a 'screenshot' and a 
'capture' and many only clicked on 'screenshots.' One potential reason 
for user's attractions to screenshots was the thumbnail that appeared on 
hover for screenshots. Additionally less than half of users noted the 
metadata text that appeared on hover over all captures. 


Task 

Analysis 



Task 

Analysis 


Capture Page 

• Users with less technical expertise tended to find the Sparkline graph 
on the capture page easier to use than the Explore page graph, as it 
was simplified and made it easier for users to focus on selecting a 
specific date. Additionally, however, most users were unable to select 
the specific date they intended to select using the Sparkline. 

• Only one user correctly identified the Sparkline graph as representing 
the overall number of captures 

• Users were unsure of when the page had finished loading, especially in 
cases where some media could not be rendered. 

• Some users thought that the capture was a 'screenshot' and did not 
appear to see the hyperlinks in the loaded capture page. Those who 
did notice the hyperlinks were very skeptical of if the links would work. 

• Users liked having the search box on the capture page, unfortunately 
many users were unable to properly format their query (URL). 

• Users who attempted to navigate back to the Wayback beta site by 
clicking on the Wayback logo on the capture page were sent to a 
petabox error page. 



Key Insights 
and Findings 


Navigation Bar 

The global Archive.org navigation bar was confusing and distracting for 
users. The navigation bar often led users to believe that searches could be 
filtered by media type. The exposed search box also caused problems for 
some users. 

Recommendation 


fffij Internet Archive v | Wayback Machine: Search the web as it was 


LOG IN Search 


mDacuittiP 


A Minimal Approach to a Universal Navigation Bar 


Create a minimized, breadcrumb based navigation bar to connect all of 
the Internet Archive's domains would allow for users to continue to explore 
and discover the many projects that are a part of the Internet Archive 
without creating a navigational bias towards Archive.org domains. The 
goal of this design is to place the navigational focus primarily on Ul 
elements related to the Wayback Machine, while still allowing for free 
movement into Archive. org's holdings. 








Key Insights 
and Findings 
Continued 


Homepage 

Users had trouble deciphering what the Wayback Machine was from the 
homepage, were often distracted by the Tools and Archive-it sections and 
found the Wayback logo to be distracting or a sign of untrustworthiness. 

Recommendation 


Internet Archive v | WayBack Machine 



About How it works 


urt the web as it wa 

Explore 472 billion web pages saved over time. 



About Save Page Now 

The WayBack Machine is a digital archive of Capture a web page as it appears now 

the internet, composed of billions of web for use as a trusted citation in the future 

pages that have been saved over time. 

Using the WayBack Machine you can view how a URL appeared in the past. 1 P" 

The WayBack Machine is not comprised of screenshots, but instead of 
interactive web captures, often with links to addltonal pages or domains. 

Using the WayBack Machine you can view how a URL appeared in the past. 

The WayBack Machine is not comprised of screenshots, but instead of 
interactive web captures, often with links to additonal pages or domains. 


Featured Historical Captures of Web Pioneers 



www.yahoo.com 

December. 1996 


favorite book. 
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www.goodreads.com 

www.geocities.com 



WayBack Tools and APIs 


Build your own tools 


Banish broken links from your blog 


Archive-it 


FAQ 


Archive-it enables you to capture, manage and 
search collections of digital content without any 
technical expertise or hosting facilities. 


Norl grape silver beet broccoli kombu beet 
greens fava bean potato 


Help users get where they were going 


Bring the WayBack Machine with you 


Donate 


Feedback 


Nori grape silver beet broccoli kombu beet 
greens fava bean potato 


Nori grape silver beet broccoli kombu beet 
greens fava bean potato 


Help Terms of Service Contact Us 


Placing an emphasis on the search bar 
as well as some short, textual 
introductions to the Wayback Machine 
could help to set user expectations of 
the Wayback Machine as well as help 
users figure out how to utilize the 
website. 

Additionally, advanced tools and 
services would be located lower on the 
page and deemphasized. 


Grouping Information on the Homepage 






Presentation of Results 


Many users wanted more information about each of the results in the 
result list, primarily: URL, website title, description and an overview of 
capture information (capture range, last capture). 


Creating a 'result page' where results 
were expanded upon to include the 
website's title, URL, thumbnail keywords 
and an overview of capture information 
in a format reminiscent of popular 
search engines would help users to 
determine which of the results is most 
relevant to their search. The addition of 
a mini-sparkline might also help prep 
users for the Explore page graph. 

The type to search could function as 
Google.com currently functions. 


Key Insights 
and Findings 
Continued 


Recommendation 


Internet Archive v | Wayback Machine 


uiayBacttciio 


cat army of doom 


6 results found 
Suggested searches: 








www.catarmyofdoom.com 
Keywords: Cat Army of Doom 

Saved 3.246 times between December 18. 1996 and May 8. 2016 


Keywords: Cat Army of Doom 

Saved 3,246 times between December 18. 1996 and May 8, 2016 


catarmyofdoom.fanfic.net 
Keywords: Cat Army of Doom 

Saved 3,246 times between December 18. 1996 and May 8, 2016 


Keywords: Cat Army of Doom 

Saved 3,246 times between December 18. 1996 and May 8. 2016 


Keywords: Cat Army of Doom 

Saved 3,246 times between December 18, 1996 and May 8. 2016 


Keywords: Cat Army of Doom 


Help Terms of Service Contact Us 


Peanut Scallion Spinach Tomato 

Turnip greens yarrow ricebean rutabaga endive cauliflower sea 
9 kohlrabi amaranth water spinach avocado daikon napa 
e asparagus winter purslane kale. Celery potato scallion 
desert raisin horseradish spinach carrot soko. Lotus root water 
spinach fennel kombu maize bamboo shoot green bean swiss chard 
seakale pumpkin onion chickpea gram corn pea. Brussels sprout 
coriander water chestnut gourd swiss chard wakame kohlrabi 
beetroot carrot watercress. 


Expanded Results 





Key Insights 
and Findings 
Continued 


Explore Graph 

Many users had trouble deciphering the Explore graph and were unsure of 
how to use the graph to learn about their query or locate a specific capture 
of the website they searched for. 


Recommendation 


www.kickstarter.com/catarmyofdoom 


Captures First Capture Last Capture 





Thursday, 28 of Oct 1997 
14:11:15 GMT 


Why was this captured? 

© ® © 

Web 51 crawl alexacrawis 


»• 

• • 


Updated Graph with Added Information on Hover 


Adding additional information related to the graph both in a header and 
throughout the graph itself could help users learn the elements of the 
Explore graph more rapidly. Hovers would be added to explain the graph 
labels and highlight specific captures upon hover. 




Key Insights 
and Findings 
Continued 


Feedback from the System 

Users had a hard time understanding many of the error messages they 
encountered while using the Wayback Machine (robots.txt, 504 errors, etc). 
Additionally, users were often unsure of if a page had completed loading 
or if their next selection was beginning to load. 

Recommendation 

Increasing the amount of 'human-friendly' information in error messages, 
as well as a redesign of the error message pages to fit the updated look 
and feel of the Wayback Machine would help users feel empowered to 
read error messages and therefore differentiate between bugs and 
restrictions set by a domain due to robots.txt. 

Adding additional feedback from the system (progress bars or loading 
indicators) could help relieve user frustration. A way to express that a 
capture is loaded in its entirety would help users understand that some 
media might not be available in a specific capture. Additionally hover 
information related to which links in a capture are 'live' could entice users 
to spend more time exploring captures. 



Questions, comments, or concerns? 

Please don't hesitate to reach out to me on Slack, 
in person or by email at carolyn@archive.org 



